Repository navigation
Merge latest changes from main to 'Documentation' branch - #192
Open
rsareddy0329 wants to merge 256 commits into
Open
rsareddy0329 wants to merge 256 commits into
rsareddy0329 wants to merge 256 commits into
Conversation
… regionalized HMA URI (#141)
* Add unique time string to integ test * Update syntax
* Training CLI & SDK: example notebook and README update * Update training cli example notebook --------- Co-authored-by: Roja Reddy Sareddy <rsareddy@amazon.com>
* Update inferenece SDK examples * Update readme
* Enable Hyperpod telemetry * Enable Hyperpod telemetry * Enable Hyperpod telemetry * Enable Hyperpod telemetry * Enable Hyperpod telemetry * Enable Hyperpod telemetry * CLI: Enable Telemetry * CLI: Enable Telemetry --------- Co-authored-by: Roja Reddy Sareddy <rsareddy@amazon.com>
* update help text to avoid truncation * update volume flag to support hostPath and pvc, before e2e testing * clean up and e2e working * Minor updates after PR * update * Added unit tests for volume, all cli unit tests passed
Co-authored-by: pintaoz <pintaoz@amazon.com>
* Update inference config and integ tests * Update integ tests for new canaries
* Manual release v3.0.2 * Update changelog --------- Co-authored-by: pintaoz <pintaoz@amazon.com>
* Update readme for volume flag * Add schema pattern check to pytorch-job template, unit test added, all test passed locally
…8s (#138) * Add k8s version validation check between server and client version according to the supported versioning constraints by k8s * Fix unit test cases * Move regex to a constant. **Description** - Removed an integration test case that was being mocked. - Moved a regex to a constant. **Testing Done** Unit test cases pass no changes made to integration test cases and they should not be affected. * Add k8s version validation check between server and client version according to the supported versioning constraints by k8s * Add ref link for version comptability contraints **Description** Added a link to k8s documentation mentioning the constraints that rule the version compatibility of client and server. **Testing Done** No breaking changes.
* Fix SDK training test: Add wait time before refresh * Fix training tests in canaries
…189) Co-authored-by: pintaoz <pintaoz@amazon.com>
* Update documentation-with-new-changes branch with latest changes from main (#190) * Fix training test (#184) * Fix SDK training test: Add wait time before refresh * Fix training tests in canaries * Update logging information for submitting and deleting training job (#189) Co-authored-by: pintaoz <pintaoz@amazon.com> --------- Co-authored-by: Zhaoqi <zhaoqiwang.baruch@gmail.com> Co-authored-by: pintaoz-aws <167920275+pintaoz-aws@users.noreply.github.com> Co-authored-by: pintaoz <pintaoz@amazon.com> * Documentation Fixes (#191) Co-authored-by: Roja Reddy Sareddy <rsareddy@amazon.com> * update documentation with new changes branch with latest changes (#194) * Fix training test (#184) * Fix SDK training test: Add wait time before refresh * Fix training tests in canaries * Update logging information for submitting and deleting training job (#189) Co-authored-by: pintaoz <pintaoz@amazon.com> --------- Co-authored-by: Zhaoqi <zhaoqiwang.baruch@gmail.com> Co-authored-by: pintaoz-aws <167920275+pintaoz-aws@users.noreply.github.com> Co-authored-by: pintaoz <pintaoz@amazon.com> * Documentation Fixes (#195) * Documentation Fixes * Documentation Fixes --------- Co-authored-by: Roja Reddy Sareddy <rsareddy@amazon.com> * Documentation Fixes (#197) * Documentation Fixes * Documentation Fixes * Documentation Fixes * Documentation Fixes --------- Co-authored-by: Roja Reddy Sareddy <rsareddy@amazon.com> * Documentation Fixes (#198) * Documentation Fixes * Documentation Fixes * Documentation Fixes * Documentation Fixes * Documentation Fixes --------- Co-authored-by: Roja Reddy Sareddy <rsareddy@amazon.com> * Documentation fixes (#199) * Documentation Fixes * Documentation Fixes * Documentation Fixes * Documentation Fixes * Documentation Fixes * Documentation Fixes --------- Co-authored-by: Roja Reddy Sareddy <rsareddy@amazon.com> --------- Co-authored-by: Zhaoqi <zhaoqiwang.baruch@gmail.com> Co-authored-by: pintaoz-aws <167920275+pintaoz-aws@users.noreply.github.com> Co-authored-by: pintaoz <pintaoz@amazon.com> Co-authored-by: Roja Reddy Sareddy <rsareddy@amazon.com>
…s to view SDK config code (#188) Co-authored-by: Mohamed Zeidan <zeidmo@amazon.com>
* Add instance type support for ml.p6e-gb200.36xlarge Updated support for ml.p6-b200.48xlarge as well * Add ml.p6e-gb200.36xlarge to efa plugin
…holder value (#206) Co-authored-by: Mohamed Zeidan <zeidmo@amazon.com>
Co-authored-by: Roja Reddy Sareddy <rsareddy@amazon.com>
* Add labels to the top level metadata (#158) Co-authored-by: pintaoz <pintaoz@amazon.com> * Implemented GPU Quota Allocation Feature. Co-authored-by: aleszewi <aleszewi@amazon.com> * Revert "Implemented GPU Quota Allocation Feature." This reverts commit 790b8f1df59494a982463aaed9e5b3f2afa44123. * Fix: Template issue - pick user defined template version (#154) * Fix: Template issue - pick user defined template version * Fix: Template issue - pick user defined template version & add topology labels in 1.1 * Fix: Template issue - pick user defined template version & add topology labels in 1.1 --------- Co-authored-by: Roja Reddy Sareddy <rsareddy@amazon.com> * Fix: Add __init__ to the new schema (#163) * Fix: Template issue - pick user defined template version * Fix: Template issue - pick user defined template version & add topology labels in 1.1 * Fix: Template issue - pick user defined template version & add topology labels in 1.1 * Fix: Add __init__ to load the new schema --------- Co-authored-by: Roja Reddy Sareddy <rsareddy@amazon.com> * Add labels and annotations to top level metadata v1.1 (#165) * Add labels to top level metadata v1.1 * Move topology labels to annotations * Update topology parameter names * Add unit test --------- Co-authored-by: pintaoz <pintaoz@amazon.com> * Added GPU quota allocation. Co-authored-by: aleszewi <aleszewi@amazon.com> * Changed neuron key to neurondevice. (#177) Co-authored-by: Marta Aleszewicz <aleszewi@amazon.com> * fix: Renamed memory-in-gib to memory for consistency. (#179) cr: https://code.amazon.com/reviews/CR-214599587 Co-authored-by: Marta Aleszewicz <aleszewi@amazon.com> * Add validation to topology labels (#178) * Add validation to topology labels * Add validation to topology labels * Add validation to topology labels --------- Co-authored-by: Roja Reddy Sareddy <rsareddy@amazon.com> * Add integ tests for topology annotations (#180) * Add labels to top level metadata v1.1 * Move topology labels to annotations * Update topology parameter names * Add unit test * Topology integ tests * Add invalid test case * Add empty test case --------- Co-authored-by: pintaoz <pintaoz@amazon.com> * Add integration tests for gpu quota allocation feature (#184) * add integration tests for gpu quota allocation feature * add valueError assertions for invalid test cases * Updating the CHANGELOG and minor version --------- Co-authored-by: pintaoz-aws <167920275+pintaoz-aws@users.noreply.github.com> Co-authored-by: pintaoz <pintaoz@amazon.com> Co-authored-by: Marta Aleszewicz <aleszewi@amazon.com> Co-authored-by: rsareddy0329 <rsareddy0329@gmail.com> Co-authored-by: Roja Reddy Sareddy <rsareddy@amazon.com> Co-authored-by: mx26pol <martacalif@gmail.com> Co-authored-by: satish Kumar <satishgollaprolu@gmail.com>
…gs to hyp-jumpstart-endpoint (#213) * Update generate_click_command inject logic to not expose unwanted flags to hyp-jumpstart-endpoint * Update unit tests for bug fix, change --label_selector to --label-selector
* Update generate_click_command inject logic to not expose unwanted flags to hyp-jumpstart-endpoint * Update unit tests for bug fix, change --label_selector to --label-selector * Update README, example notebooks and documentation to 1)remove model_version, 2)add --model-volume-mount-name 3)remove tar.gz from --model-location 4)update unique mount_path for --volume * Update README, example notebooks and documentation to remove tls-config for jumpstart * minor update to remove tar.gz from --model-location for documentation
#219) * add metadata_name argument to js and custom endpoint to match with SDK * fix integ
* Bug fix in Hyperpod recipe rendering * Bug fix in Hyperpod recipe rendering --------- Co-authored-by: Roja Reddy Sareddy <rsareddy@amazon.com>
* docs: add v3.10.0 changelog * chore: bump project version to 3.10.0
…monSets (#451) Under node DiskPressure, HMA was evicted moments before a GPU failure went undetected and unremediated (P438078245). Setting priorityClassName to the built-in system-node-critical makes HMA the last pod evicted (after training and other monitoring pods), preserving health monitoring during node pressure. Mirrors what EKSNodeMonitoringAgent already does. Applied to both the health-monitoring-agent (NVIDIA) and health-monitoring-agent-non-nvidia (Trainium/Inferentia) DaemonSets. Configurable via .Values.priorityClassName, defaulting to system-node-critical.
* fix: update instance type to g6e for deepseek integ tests * fix: use smaller model so tests run on g5.8xlarge --------- Co-authored-by: Syed Jafri <syedjfr@amazon.com>
Bumps the inference-operator Helm chart to 2.5.0 (operator v3.5), updates the umbrella chart dependency pin, and adds the weights-cache node affinity field to the inference-operator CRDs.
* fix: conditional aws-hyperpod namespace creation * fix: deep health check namespace reference * chore: allow conditional creation of mpi-operator chart * chore: address review changes
* Add v3.11.0 changelog and version bump * Release 3.11 - update setup.py --------- Co-authored-by: Syed Jafri <syedjfr@amazon.com>
… load_kubeconfig (#458)
- Chart 2.5.0 -> 2.6.0, appVersion 3.5 -> 3.6, image tag v3.5 -> v3.6 - Regenerated IEC/JSM CRDs: add spec/status.inferenceGateway opt-in (CR-295860367) and IEC spec.kubernetes.podTemplateAnnotations (CR-299953131). Additive, backward compatible. Co-authored-by: Xuan Lu <xua@amazon.com>
…#463) Co-authored-by: Xuan Lu <xua@amazon.com>
* feat(helm): bump inference-operator to v3.7, add priorityClassName, CloudWatch logging, tolerations, and MP mode * fix: restore trailing newline in values.yaml * feat(helm): add fluentbit ConfigMap, registryDomain resolver, bump Chart to v2.7.0/appVersion 3.7, bump fsx to 1.17.0
Bump the HyperPod gpu-operator subchart to v26.7.0 to clear out-of-SLA container CVEs on the v26.3.1 stack. Pairs with the ECR replication of the v26.7.0 family; toolkit stays default-enabled for coexistence. - operator + validator v26.3.1 -> v26.7.0 - device-plugin + gfd v0.19.0 -> v0.20.0 - container-toolkit v1.19.0 -> v1.20.0 - mig-manager v0.14.0 -> v0.15.0 Co-authored-by: Stephen Via <svia@amazon.com>
…with bug fixes and CVE fixes (#468) Bug Fixes * Telemetry-only health check timeouts no longer trigger node reboots * Raised the driver-issue timeout threshold from 3 to 5 consecutive checks * Added a 20s component timeout for the GPU fabric manager check New Configuration * XID_63_ACTION_DISABLED (default false). When set to true, Xid 63 (GPU row remap pending) is recorded as a warning with no reboot action. Detection and metrics are unaffected; only the remediation action is suppressed. Security * Go dependency updates for CVE remediation.
* fix: Stabilize integration cluster tests * fix: Isolate recipe test storage * fix: Support Kubernetes 37 models
fix(inference-operator): update Helm to EKS add-on migration script.
…#449) The kubeflow training-operator Deployment has no tolerations, so it cannot be scheduled onto nodes carrying the HyperPod deep health check taint sagemaker.amazonaws.com/node-health-status=Unschedulable:NoSchedule. On clusters where every node is tainted during an on-start deep health check, the operator pod stays Pending and PyTorchJob-based checks such as MultiNodeTraining never get reconciled. Other controllers and device plugins in this chart already tolerate the same taint (mpi-operator, neuron-device-plugin, nvidia-device-plugin, aws-efa-k8s-device-plugin). This makes training-operators consistent.
`hyp create` rendered standard project template manifests (k8s.jinja / cfn_params.jinja) with a plain `jinja2.Template`. Rendering now uses `jinja2.sandbox.ImmutableSandboxedEnvironment` so template expressions cannot access unsafe Python attributes or callables. The recipe rendering path (recipe_utils._render_k8s_template) already used a sandboxed environment; this aligns the standard-template path in _create_standard_template with it. Adds tests asserting that a disallowed template expression is blocked and that benign templates still render. Co-authored-by: Mufaddal Rohawala <mufi@amazon.com>
hyp list cluster-stack and HpClusterStack.list() return all CloudFormation stacks in the account and Region, but the help text, output header, and documentation described them as HyperPod-only. Clarify the behavior in the CLI help, SDK docstring, output header, README, Read the Docs, and example notebooks. Also fix the invalid `hyp list hyp-cluster` usage example and add regression tests covering mixed HyperPod and unrelated stacks plus help/behavior consistency. Refs #348
Co-authored-by: Roja Reddy Sareddy <rsareddy@amazon.com>
…default (#479) * feat: add hyperpod-kubernetes-resource-monitor subchart, disabled by default Adds the HyperPod Kubernetes Resource Monitoring Agent as the hyperpod-kubernetes-resource-monitor subchart (enabled: false in the parent values). * feat: name hyperpod-kubernetes-resource-monitor resources hyperpod-kubernetes-resource-monitoring-agent Use fixed resource names, like health-monitoring-agent: the Deployment, ServiceAccount, ClusterRole and ClusterRoleBinding are named hyperpod-kubernetes-resource-monitoring-agent, whatever the release name. The values key (hyperpod-kubernetes-resource-monitor) and the app=cluster-monitoring-agent label are unchanged. --------- Co-authored-by: Xin Wang <xwnamz@amazon.com>
* fix: remove NCCL RAS settings the resource monitor ignores The agent's NCCL RAS job monitor takes each job's endpoints from the pod IPs of its training pods. It never reads ncclRas.nodes, autoDiscover or nodeSelector, and it does not use retryDelay or outputDir, so setting them had no effect. Remove them from the values and the Deployment env. maxRetries is the number of connection attempts per endpoint in each monitoring cycle; the comment now says so. Older agent images read the removed env vars with the same defaults the chart set, so their behavior does not change. * fix: pin hyperpod-kubernetes-resource-monitor to image 1.0.508.0_1.0.327.0 The new image fixes the job-level NCCL RAS state, which was always no_data, and stops querying the training operator's pod-manager pods as NCCL RAS endpoints. --------- Co-authored-by: Xin Wang <xwnamz@amazon.com>
Turn the hyperpod-kubernetes-resource-monitor subchart on by default, like health-monitoring-agent. Set hyperpod-kubernetes-resource-monitor.enabled=false to skip it. The SageMaker console already sends the value explicitly, so console installs are unchanged. Co-authored-by: Xin Wang <xwnamz@amazon.com>
- Chart 2.7.0 -> 2.8.3, appVersion 3.7 -> 3.8, image tag v3.7 -> v3.8; parent chart dependency bumped to 2.8.3 - Regenerated IEC/JSM CRDs: add IEC spec.dynamoSpec (managed NVIDIA Dynamo), status.selfHealCount, and path-safe pattern and maxLength on invocationEndpoint and loadBalancer.healthCheckPath. Descriptions reworded. Backward compatible. - Manager memory limit 256Mi -> 512Mi. - RBAC: grant the operator access to nvidia.com dynamographdeployments. - Subcharts: aws-mountpoint-s3-csi-driver 1.14.1 -> 1.15.0 (with sidecar image pins), aws-fsx-csi-driver 1.17.0 -> 1.19.0; Chart.lock regenerated. Signed-off-by: Ophelia Yang <ziyng@amazon.com>
hyperpod-resource-ownership records who created each RayJob, HyperPodPyTorchJob, InferenceEndpointConfig and JumpStartModel (the created-by annotation) and lets only the creator, the operators and cluster admins update or delete the ones marked OwnerOnly, including through their status and scale subresources. The owner can change ownership-type; created-by is immutable. hyperpod-compute-configuration installs the ComputeConfiguration CRD and its admission webhook. An admin defines named compute shapes in a namespace; a RayJob, HyperPodPyTorchJob or InferenceEndpointConfig that references one by label gets its compute filled in and checked, and namespaces labelled compute.configuration.sagemaker.amazonaws.com/enforcement=enabled require every such workload to use one. Both webhook images come from the regional HyperPod ECR repository for the 14 supported Regions, by tag or by digest; an image override must be pinned by digest. Both require cert-manager and Kubernetes 1.30+. Co-authored-by: Meghana Janapareddy <meghajan@amazon.com>
This branch is waiting to be deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR Approval Steps
For Requester
For Reviewer
For Requestersection to double check each item.